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SYSTEM AND METHOD FOR COMMUNICATING THE 
SYNCHRONIZATION STATUS OF MEMORY MODULES DURING 
INITIALIZATION OF THE MEMORY MODULES 

TECHNICAL FIELD 

[001] The present invention relates to a processor-based system, and more 

particularly, to a processor-based system having a memory module with a memory hub 
coupling several memory devices to a processor or other memory access device. 

BACKGROUND OF THE INVENTION 

[002] Processor-based systems, such as computer systems, use memory 

devices, such as dynamic random access memory ("DRAM") devices, as system 
memory to store instructions and data that are accessed by a processor. In a typical 
computer system, the processor communicates with the system memory through a 
processor bus and a memory controller. The processor issues a memory request, which 
includes a memory command, such as a read command, and an address designating the 
location from which data or instructions are to be read or to which data or instructions 
are to be written. The memory controller uses the command and address to generate 
appropriate command signals as well as row and column addresses, which are applied to 
the system memory. In response to the commands and addresses, data is transferred 
between the system memory and the processor. The memory controller is often part of 
a system controller, which also includes bus bridge circuitry for coupling the processor 
bus to an expansion bus, such as a PCI bus. 

[003] Although the operating speed of memory devices has continuously 

increased, this increase in operating speed has not kept pace with increases in the 
operating speed of processors. Even slower has been the increase in operating speed of 
memory controllers coupling processors to memory devices. The relatively slow speed 
of memory controllers and memory devices limits the data bandwidth between the 
processor and the memory devices. 



[004] One approach to increasing the data bandwidth to and from memory 

devices is to use multiple memory devices coupled to the processor through a memory 
hub as shown in Figure 1. A computer system 10 using a memory hub architecture 
includes a processor 104 for performing various computing functions, such as executing 
specific software to perform specific calculations or tasks. The processor 104 includes 
a processor bus 106 that normally includes an address bus, a control bus, and a data bus. 
The processor bus 106 is typically coupled to cache memory 108, which, is typically 
static random access memory ("SRAM"). Finally, the processor bus 106 is coupled to a 
system controller 110, which is also sometimes referred to as a bus bridge. 
[005] The system controller 1 10 contains a memory hub controller 1 12 that is 

coupled to the processor 104. The memory hub controller 1 12 is also coupled to several 
memory modules 114a-n through a bus system 115. Each of the memory modules 
114a-n includes a memory hub 116 coupled to several memory devices 118 through 
command, address and data buses 1 17. The memory hub 116 efficiently routes memory 
requests and responses between the controller 112 and the memory devices 118. 
Computer systems employing this architecture can have a higher bandwidth because the 
processor 104 can access one memory module 114a-n while another memory module 
114a-n is responding to a prior memory access. For example, the processor 104 can 
output write data to one of the memory modules 114a-n in the system while another 
memory module 1 14a-n in the system is preparing to provide read data to the processor 
104. The operating efficiency of computer systems using a memory hub architecture 
can make it more practical to vastly increase data bandwidth of a memory system. A 
memory hub architecture can also provide greatly increased memory capacity in 
computer systems. 

[006] The system controller 110 also serves as a communications path to the 

processor 104 for a variety of other components. More specifically, the system 
controller 110 includes a graphics port that is typically coupled to a graphics controller 
121, which is, in turn, coupled to a video terminal 123. The system controller 110 is 
also coupled to one or more input devices 120, such as a keyboard or a mouse, to allow 



an operator to interface with the computer system 10. Typically, the computer system 
10 also includes one or more output devices 122, such as a printer, coupled to the 
processor 104 through the system controller 1 10. One or more data storage devices 122 
are also typically coupled to the processor 104 through the system controller 110 to 
allow the processor 104 to store data or retrieve data from internal or external storage 
media (not shown). Examples of typical storage devices 124 include hard and floppy 
disks, tape cassettes, and compact disk read-only memories (CD-ROMs). 
[007] Although there are advantages to utilizing a memory hub for accessing 

memory devices, the design of the hub memory system, and more generally, computer 
systems including such a memory hub architecture, becomes increasingly difficult. The 
memory modules 114a-n each internally operate synchronously although perations 
between memory modules are asynchronous. The command, address, and data signals 
transferred to the memory module 114a-n are normally latched or strobed into the 
/memory modules 114a-n by a clock signal. However, as transfer rates increase, the 
time during which the command, address and data signals as received at the memory 
hubs 116 are valid decreases. This period during which the signals are valid is 
commonly referenced by those ordinarily skilled in the art as the "window" or "eye." 
Not only does the size of the eye for command, address, and data signals decrease, but 
the time or location of the eye can also vary because of various factors, such as timing 
skew, voltage and current drive capability, and the like. In the case of timing skew of 
signals, it often arises from a variety of timing errors such as loading on the lines of the 
bus and the physical lengths of such lines. 

[008] As the size of signal eyes decrease at higher transfer rates, the variations 

in the location of the signal eyes become more of a problem. One technique to alleviate 
this problem to some extent is to couple a clock to the memory modules. As shown in 
Figure 1, a clock generator 500 generates a clock signal CLK and couples it to the 
memory hub controller 112 and each of the memory hubs 116 in respective memory 
modules 114a-n along with the command, address and data signals that are coupled 
through the bus system 115. The CLK signal is coupled from the memory hub 
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controller 112 in synchronism with the command, address and data signals so it, in 
theory, should be usable by the memory hubs 116 to define the eye during for the 
command, address and data signals as they are received at the memory hubs 116. 
However, in practice, even this approach becomes ineffective as signal transfer rates 
continue to decrease. In particular, the CLK signal may be subject to different 
conditions than the command, address and data signals, such as being coupled through a 
physically different signal path or being loaded to a greater degree. Also, for the clock 
forwarding techniques used in the computer system 10 to successfully function at higher 
clock speeds, the layout of conductors between the memory hub controller 1 12 and the 
memory hubs 116 must be precisely controlled. 

[009] One technique that has been proposed to allow the CLK signal to 

continue being used to strobe command, address and data signals at higher transfer rates 

is to include circuitry (not shown) in the memory hubs 116 that adjusts the timing of the 

CLK signal within each of the hubs 116 so that it is aligned with the signal eye. \ 

However, this technique adds a fair degree of complexity to the memory hubs 1 16 and 

is not always effective. 

[010] There is therefore a need for a system and method that allows command, 

address and data signals to be coupled between a memory hub controller and one or 
more memory hubs in respective memory modules that avoids problems of 
synchronizing a clock signal coupled between the memory hub controller and memory 
hubs along with the command, address, and data signals. 

SUMMARY OF THE INVENTION 

A memory system includes a memory hub controller coupled to a 
plurality of memory hubs each of which includes at least one receiver that is initialized 
before use. The memory hub controller transmits a downstream initialization complete 
signal to one of the memory hubs when initialization of a receiver in the memory hub 
controller has been completed. Each of the memory hubs determines if it is receiving a 
respective downstream initialization complete signal, and it also determines if 



initialization of at least one internal receiver has been completed. If so, each of the 
memory hubs except for a final memory hub furthest downstream from the memory hub 
' controller transmits a downstream initialization complete signal. If the final memory 
hub receives the respective downstream initialization complete signal and initialization 
of at least one receiver in the final memory hub has been completed, the final memory 
hub transmits an upstream initialization complete signal Each of the memory hubs 
except for the final memory hub determines if it is receiving a respective upstream 
initialization complete signal, and it also determines if initialization of all receivers in 
the memory hub has been completed. If so, the memory hub transmits an upstream 
initialization complete signal. When the memory hub controller receives a respective 
upstream initialization complete signal, it transitions the memory system to normal 
operation. 

BRIEF DESCRIPTION OF THE DRAWINGS 

[011] Figure 1 is a block diagram of a computer system that includes several 

memory modules having a memory hub architecture coupled to a memory hub 
controller. 

[012] Figure 2 is a block diagram of a computer system that includes several 

memory modules having a memory hub architecture according to one embodiment of 
the present invention. 

[013] Figure 3 is a block diagram of one embodiment of receivers and 

transmitters used in the computer system of Figure 2 or some other system in 
accordance with the present invention. 

[014] Figure 4 is a flow chart showing one embodiment of a technique for 

communicating the synchronization status of each memory module used in the 
computer system of Figure 2 during initialization so that a memory hub controller used 
in the system of Figure 2 can determine when initialization of all memory modules have 
been completed. 



[015] Figure 5 is a block diagram of a memory hub that may be used the 

memory modules that are used in the computer system of Figure 2. 

DETAILED DESCRIPTION OF THE INVENTION 

[01 6] Embodiments of the present invention are directed to a memory module 

and memory hub controller each having the capability of generating a clock signal for 
strobing data signals during the "eye" of the data signals when the data signals are valid. 
More particularly, embodiments of the present invention are directed to a system and 
method for communicating the initialization status of the memory module and memory 
hub controller. Certain details are set forth below to provide a sufficient understanding 
of various embodiments of the invention. However, it will be clear to one skilled in the 
art that the invention may be practiced without these particular details. In other 
instances, well-known circuits, control signals, and timing protocols have not been 
shown in detail in order to avoid unnecessarily obscuring the invention. 
[017] A computer system 100 having a hub memory system according to one 

embodiment of the invention is shown in Figure 2. The computer system 100 uses 
many of the same components that are used in the computer system 10 of Figure 1. 
Therefore, in the interest of brevity, these components have been provided with the 
same reference numerals, and an explanation of their the functions and operation will 
not be repeated. 

[01 8] As in the computer system 1 0 of Figure 1 , the system controller 1 1 0 also 

includes a memory hub controller 128 that is coupled to several memory modules 
130a,b...n, which serve as system memory for the computer system 100. The memory 
modules 130 are each coupled to a high-speed downstream bus 132 and a high-speed 
upstream bus 134. The downstream bus 132 extends downstream from the memory hub 
controller 128 and both upstream and downstream from each of the memory modules 
130 except the memory module 130n furthest from the memory hub controller 128. 
Similarly, the upstream bus 134 extends upstream from each of the memory modules 
130 and downstream from all of the memory modules 130 except the memory module 



130n. Each of these buses 132, 134, include a discrete data bus, although they may also 
include discrete command and address buses, a combined command/address bus, or 
some other bus system. However, the explanation of the various embodiments will be 
with respect to a data bus, it being understood that a similar technique can be used to 
strobe command and address signals. 

[019] The downstream buses 132 couple data away from the memory hub 

controller 128, and the upstream buses 134 couple data toward the memory hub 
controller 128. Therefore, the downstream bus 132 couples write data to and from each 
of the memory modules 130, except for the memory module 130n furthest downstream 
from the memory hub controller 128, which only receives write data. Similarly, the 
upstream bus 134 couples read data to and from each of the memory modules 130, 
except for the memory module 130n furthest downstream from the memory hub 
controller 128, which only transmits read data. The downstream bus 132 also couples 
write data from the memory hub controller 128, and the upstream bus 134 couples read 
data to the memory hub controller 128. Significantly, the buses 132, 134 need not 
couple clock signals to and from the memory modules 130 and the memory hub 
controller 128 for the purpose of allowing the memory modules 130 to capture data 
transmitted through the buses 132, 134. Instead, as explained in greater detail below, 
each of the memory modules 130 and the memory hub controller 128 generates signals 
internally to strobe the data coupled through the buses 132, 134. 
[020] The memory modules 130 are shown coupled to the memory hub 

controller 128 in a point-to-point coupling arrangement in which each of the buses 132 
and 134 are coupled only between two points. However, it will be understood that other 
topologies may also be used. For example, it may be possible to use a multi-drop 
arrangement in which a single downstream bus (not shown) and a single upstream bus 
(not shown) are coupled to all of the memory modules 130. A switching topology may 
also be used in which the memory hub controller 128 is selectively coupled to each of 
the memory modules 130 through a switch (not shown). Other topologies that may be 
used will be apparent to one skilled in the art. 
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[021] Each of the memory modules 130 includes a first receiver 142 that 

receives write data through the downstream bus 132, a first transmitter 144 that 
transmits read data upstream through the upstream bus 134, a second transmitter 146 
that transmits write data downstream through the downstream bus 132, and a second 
receiver 148 that receives read data through the upstream bus 134. 
[022] The memory modules 130 also each include a memory hub local 150 that 

is coupled to its first receiver 142 and its first transmitter 144. The memory hub local 
150 receives write data through the downstream bus 132 and the first receiver 142 and 
couples the write data to one or more of sixteen memory devices 160, which, in the 
example illustrated in Figure 2, are synchronous dynamic random access memory 
("SDRAM") devices. However, a fewer or greater number of memory devices 160 may 
be used, and memory devices other than SDRAM devices may also be used. The 
memory hub local 150 is coupled to each of the memory devices 160 through a bus 
system 164, which normally includes a control bus, an address bus, and a data bus. 
However, other bus systems, such as a bus system using a shared command/address bus, 
may also be used. 

[023] The memory hub 150 also receives read data from one or more of the 

memory devices 160 and couples the read data through the first transmitter 144 and the 
upstream bus 134. In the event the write data coupled through the downstream bus 132 
and the first receiver 142 is not being directed to the memory devices 160 in the 
memory module 130 receiving the write data, the write data are coupled though a 
downstream bypass path 170 to the second transmitter 146 for coupling through the 
downstream bus 132. Similarly, if read data is being transmitted from a downstream 
memory module 130, the read data is coupled through the upstream bus 134 and the 
second receiver 148. The read data are then coupled upstream through an upstream 
bypass path 174, and then through the first transmitter 144 and the upstream bus 134. 
The second receiver 148 and the second transmitter 146 in the memory module 130n 
furthest downstream from the memory hub controller 128 are not used and may be 
omitted from the memory module 130n. 
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[024] The memory hub controller 128 also includes a transmitter 180 coupled 

to the downstream bus 132, and a receiver 182 coupled to the upstream bus 134. The 
downstream bus 132 from the transmitter 180 and the upstream bus 134 to the receiver 
182 are coupled only to the memory module 130a that is the furthest upstream to the 
memory hub controller 128. The transmitter 180 couples write data from the memory 
hub controller 128, and the receiver 182 couples read data to the memory hub controller 
128. 

[025] The computer system 100 also includes a reference clock generator 190, 

which generates a reference clock signal that is coupled to the memory hub controller 
128 and each of the memory modules 130. The memory hub controller 128 and the 
memory modules 130 use the reference clock to generate an internal clock signal that, 
inithe embodiment of Figure 2, has a frequency of four times the frequency of the 
reference clock signal. This internal clock signal is used as a transmit clock to strobe 
data from the transmitters 144, 146, 180. The receivers 142, 148, 182 adjust the phase 
of the internal clock signal to generate a receive clock signal that is used to strobe data 
into the receivers 142, 148, 182. Briefly, the receivers 142, 148, 182 perform this 
function by receiving a known data pattern from a transmitter 144, 146, 180 to which it 
is coupled, and attempt to capture that data pattern by strobing the data as the phases of 
the receive clock signals are incrementally varied. The phase of the receive clock signal 
that best captures the data pattern is then used to strobe data into the receivers 142, 148, 
182 in normal operation. 

[026] One embodiment of the receivers 142, 182 and the transmitters 144, 180 

in the memory hub controller 128 and in one of the memory modules 130 is shown in 
Figure 3. In both cases, a receiver 200 functions as both receivers 142, 148 in the 
memory module 130 and the receiver 182 in the memory hub controller 128, and a 
transmitter 210 functions as both transmitters 144, 146 in the memory module 130 as 
well as the single transmitter 180 in the memory hub controller 128. The transmitter 
210 includes a pattern generator 220 that generates a predetermined pattern of data bits, 
and a transmit interface control 224 that controls the transmitting of the pattern. In the 
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embodiment of Figure 3, the same pattern is transmitted on all of the data bits of the 
buses 132, 134. During an initialization phase of the receiver 200 and transmitter 210, 
the transmitter 210 continuously repeats its transmission of the data pattern. 
[027] As previously explained, the receiver 200 receives the data bits from the 

transmitter 210 and strobes them in using a receive clock signal generated from the 
clock signal received from the clock generator 500 and having four times the frequency 
of the core clock. More specifically, in one embodiment of the invention, the pattern 
transmitted by the transmitter 210 is the following 32-bit pattern divided into four 
cycles each having 8 bits: "01011011 11000101 10010011 00101100" (hex 
"5BC5932C"). The data bit pattern is transmitted from right to left. In the embodiment 
of Figure 3, a bit is strobed into the receiver 200 on each transition of the receive clock 
signal, so two bits are captured by the receiver 200 on each receive clock cycle. Since 
the receive clock has a frequency of four times the core clock, eight bits of data are 
captured during each cycle of the core clock. 

[028] In the embodiment of Figure 3, the first bit is always captured on the 

positive edge of the receive clock signal. As a result, there are 16 possible patterns of 
valid data captured by the receiver 200, namely, the transmitted 32-bit pattern shifted by 
two bits for each pattern. An expected pattern memory 230 stores all 16 of these 
possible patterns, which, as previously explained, consists of eight bits. 
[029] It is significant, for reasons that will be explained below, that the 

transmitters 210 can transmit the inverse of the above 32-bit pattern, namely "10100100 
001 11010 01 101 100 1 101001 1," and the receivers 200 can detect that pattern as well as 
the fact the inverse of the pattern is being transmitted. 

[030] In the embodiment of Figure 3, a pattern comparator 234 performs three 

comparisons. First, it checks all of the data bits of the bus 132 to ensure that they all 
have the same value as each data bit is captured since the same data are transmitted on 
each data bit of the bus 132. This same comparison is performed on bus 134. 
[031] In the second comparison, the pattern comparator 234 compares the eight 

data bits captured in the receiver 200 for each core cycle to the sixteen valid 8-bit data 
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bit patterns stored in an expected pattern memory 230. For purposes of this 
comparison, it can use any of the 32 bits captured on each transition of the receive clock 
signal since the first comparison confirmed that all 32 bits were the same. Based on this 
. comparison, phase adjustment logic 240 adjusts the phase of the receive clock signal so 
that it can best capture the data coupled to the receiver. More specifically, the pattern 
comparator 234 compares the 8 bits received during any core cycle to the 16 valid 
patterns and their inverses stored in the expected pattern memory 230 to adjust the 
phase of the receive clock signal The above operation is controlled by a receive 
interface controller 244. One embodiment of the receivers 200 is shown and described 
in greater detail, and the operation of the transmitters 210 and receivers are described in 
greater detail, in U.S. Patent Application, Serial No. * having a common inventor, 
which is incorporated herein by reference. 

.[032] In the third comparison, the pattern comparator 234 checks an additional 

33 rd bit, which functions as a control bit. The pattern that is sent on the beses 132, 134 
is also sent on the control bit for each of these buses. The eight bits captured on one 
core clock is compared in the same manner as the second comparison. 
[033] The duration of the initialization process during which the phase of the 

receive clock signal is adjusted will typically be different for each of the receivers 200 
used in the memory modules 130. It is important for the receivers 200 to be able to 
provide an indication of when the adjustment of the phase of the receive clock signal 
has been completed for a least two reasons. First, once initialization of a receiver 200 
has been completed, the transmitter 210 no longer needs to transmit the 32-bit 
initialization patterns. By communicating its initialization status, the receiver 200 can 
inform the transmitter 210 when it can discontinue transmitting the 32-bit pattern. 
[034] The second reason why it is important for the receivers 200 to be able to 

communicate their respective synchronization status is because the computer system 
100 (Figure 2) cannot begin normal operation until initialization of all of the receivers 
200 in the memory modules 130 have been completed. Once all of the memory 
modules 130 have communicated their initialization completion to the memory hub 
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controller 112 (Figure 2), the memory hub controller 112, and the entire computer 
system 100, can begin normal operation. 

[035] One embodiment of a technique by which the receivers 200 can 

communicate their respective initialization status will now be explained with reference 
to the flow chart of Figure 4. The initialization process is entered at step 300, and the 
transmitter 180 (Figure 2) in the memory hub controller 128 begins transmitting test 
data, ie. 9 the 32-bit pattern "01011011 11000101 10010011 00101 100," at step 304. 
Each memory module 130 checks at step 310 to determine if its receiver 142 that is 
coupled to the downstream bus 132 is receiving test data. When the receiver 142 in a 
memory module begins receiving test data, the transmitter 144 coupled to the upstream 
bus 134 begins transmitting test data at step 314. Also, the detection of test data on the 
downstream bus 132 at step 310 causes the transmitter 146 coupled to each downstream 
bus 132 to begin transmitting test data at step 318. Thus, in a system 100 with multiple 
memory modules, the transmission of test data by the memory hub controller 128 at 
step 304 causes each downstream memory module 130 to successively transmit test data 
both upstream and downstream until both of the transmitters 144, 146 in all of the 
memory modules 130 are transmitting test data. The receiver 182 in the memory hub 
controller 128 and the receivers 142, 148 in each of the memory modules 130 then use 
the test data to adjust the phase of their respective internally generated receive clock 
signals, as previously explained. 

[036] The procedure shown in Figure 4 then checks at step 320 to determine 

when adjustment of the phases of the receive clock signals generated by the receiver 
182 in the memory hub controller 128 or the receivers 142, 148 in each memory module 
130 has been completed. When phase adjustment completion is detected at step 320, 
the transmitter 180 in the memory hub controller 128 or the "downstream" transmitter 
146 in each memory module 130 checks at step 324 to determine if the original test data 
(i.e., non-inverted test data) are still being received through the downstream bus 132. If 
a determination is made at step 324 that the original test data is not being received, the 
"downstream" transmitter 180 or 132 in the memory hub controller 128 or memory 
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module 130, respectively, inverts the test data it transmits on the downstream bus 132 at 
step 328. 

[037] There is no receiver coupled to the downstream bus 132 for the memory 

hub controller 128 so the determination at step 324 will always result in the transmitter 
180 inverting the test data at step 328 as soon as initialization of the receiver 182 has 
been completed. However, the "downstream" transmitters 146 in the respective 
memory modules 130 will not couple inverted test data to the downstream bus 132 at 
step 328 until detection of initialization completion of both receivers 142, 148 is 
detected at step 320 and receipt of inverted test data on the downstream bus 132 is 
detected at step 324. As a result, when the memory hub controller 128 or each of the 
memory modules 130 transmits inverted test data on the downstream bus 132, it notifies 
the downstream memory module that initialization has been completed. Each memory 
module 130 thus transmits inverted data downstream in sequence upon receipt of 
inverted test data from either the adjacent memory hub controller 128 or the upstream 
adjacent memory module 130 as long as initialization of both of its receivers have been 
completed. When the furthest downstream memory module 130 receives inverted test 
data on the downstream bus 132 at step 324, all of the memory modules 130 have been 
notified that initialization has been completed for all upstream memory modules and the 
memory hub controller 128. 

[038] Normal operation cannot begin responsive to the furthest downstream 

memory module 130 receiving inverted test data on the downstream bus 132 because 
the memory hub controller 128 and the memory modules 130 have not been notified 
that initialization has been completed for all of the downstream memory modules 130. 
Therefore, the process checks at step 330 to determine if inverted test data have been 
received through any upstream data bus 134. The memory module 130 furthest 
downstream from the memory hub controller 128 has no upstream data bus 134, so the 
"upstream" transmitter 144 in the furthest downstream memory module begins 
transmitting inverted test data upstream to the upstream memory module 130 at step 
336. In like manner, each "upstream" receiver 148 checks at step 330 to determine if it 
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is receiving inverted test data. If so, the upstream transmitter 144 in the memory 
module 130 likewise begins transmitting inverted test data upstream at step 336. When 
the receiver 182 in the memory hub controller 128 detects inverted test data at step 340, 
it is notified that initialization in both receivers 142, 148 in all of the memory modules 
130 have been completed. The memory hub controller 128 then switches to the normal 
operating mode at step 344 by sending NOP's, or "No Operations" on the downstream 
bus 132. 

[039] In an alternative embodiment, the completion of initialization for only 

the downstream receiver 142 is checked at step 320 so that the downstream transmitter 
146 in each memory module 130 will begin transmitting inverted test data at step 328 as 
soon as inverted test data is detected on the downstream bus 132 at step 324. In this 
embodiment, when the furthest downstream memory module detects inverted test data, 
it only signifies that initialization of the downstream receivers 142 have been 
completed. However, in this embodiment, the initialization status of the downstream 
receivers 148 are checked at step 320 along with the determination of whether inverted 
test data is being received by the "upstream" receiver 148 in the memory module. If 
inverted test data is being received through the upstream receiver 148 and initialization 
of the upstream receiver has been completed, the upstream transmitter 144 begins 
transmitting inverted test data at step 336. Therefore, receipt of inverted test data by the 
receiver 182 in the memory hub controller 128 still signifies that initialization for both 
receivers 142, 148 in all of the memory hubs 130 have been completed. 
[040] Although the transmission and detected of inverted test data provides one 

means of communicating the initialization status of the memory hub controller 128 and 
the memory modules, other communication protocols may alternatively be used. 
[041] Figure 5 shows an embodiment of the memory hub local 150 according 

to the present invention, which can be used in the memory modules 130 of Figure 2. 
The memory hub local 150 includes two input bus interfaces 410a,d, which may be used 
to couple data into the memory hub local 150, and two output bus interfaces 412a,b, 
which may be used to couple data from the memory hub the memory hub local 150. 
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[042] The bus interfaces 410a,b, 412a,b are coupled to a switch 460 through a 

plurality of bus and signal lines, represented by buses 414. The buses 414 are 
conventional, and include a write data bus and a read data bus, although a single bi- 
directional data bus may alternatively be provided to couple data in both directions 
through the bus interfaces 410a,b, 412a,b. It will be appreciated by those ordinarily 
skilled in the art that the buses 414 are provided by way of example, and that the buses 
414 may include fewer or greater signal lines, such as further including a request line 
and a snoop line, which can be used for maintaining cache coherency. 
[043] The switch 460 is coupled to four memory interfaces 470a-d which are, 

in turn, coupled to the memory devices 160 (Figure 2). By providing a separate and 
independent memory interface 470a-d for each set of memory devices 160, the memory 
hub local 150 avoids bus or memory bank conflicts that typically occur with single 
channel memory architectures. The switch 460 is coupled to each memory interface 
through a plurality of bus and signal lines, represented by buses 474. The buses 474 
include a write data bus, a read data bus, and a request line. However, it will be 
understood that a single bi-directional data bus may alternatively be used instead of a 
separate write data bus and read data bus. Moreover, the buses 474 can include a 
greater or lesser number of signal lines than those previously described. 
[044] In an embodiment of the present invention, each memory interface 

470a-d is specially adapted to the memory devices 160 to which it is coupled. More 
specifically, each memory interface 470a-d is specially adapted to provide and receive 
the specific signals received and generated, respectively, by the memory devices 160 to 
which it is coupled. Also, the memory interfaces 470a-d are capable of operating with 
memory devices 160 operating at different clock frequencies. As a result, the memory 
interfaces 470a-d isolate the processor 104 from changes that may occur at the interface 
between the memory hub 130 and memory devices 160 coupled to the memory 
hub local 150, and it provides a more controlled environment to which the memory 
devices 160 may interface. 
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[045] The switch 460 coupling the bus interfaces 410a,b, 412a,b and the 

memory interfaces 470a-d can be any of a variety of conventional or hereinafter 
developed switches. For example, the switch 460 may be a cross-bar switch that can 
simultaneously couple bus interfaces 410a,b, 412a,b to each other to provide the 
downstream bypass path 170 and the upstream bypass path 174 shown in Figure 2. The 
switch 460 can also be a set of multiplexers that do not provide the same level of 
connectivity as a cross-bar switch but nevertheless can couple the some or all of the bus 
interfaces 410a,b, 412a,b to each of the memory interfaces 470a-d. The switch 460 may 
also includes arbitration logic (not shown) to determine which memory accesses should 
receive priority over other memory accesses. Bus arbitration performing this function is 
well known to one skilled in the art. 

[046] With further reference to Figure 5, each of the memory interfaces 470a-d 

includes a respective memory controller 480, a respective write buffer 482, and a 
respective cache memory unit 484. The memory controller 480 performs the same 
functions as a conventional memory controller by providing control, address and data 
signals to the memory devices 160 to which it is coupled and receiving data signals 
from the memory device 160 to which it is coupled. However, the nature of the signals 
sent and received by the memory controller 480 will correspond to the nature of the 
signals that the memory devices re adapted to send and receive. The cache memory unit 
484 includes the normal components of a cache memory, including a tag memory, a 
data memory, a comparator* and the like, as is well known in the art. The memory 
devices used in the write buffer 482 and the cache memory unit 484 may be either 
DRAM devices, static random access memory ("SRAM") devices, other types of 
memory devices, or a combination of all three. Furthermore, any or all of these 
memory devices as well as the other components used in the cache memory unit 484 
may be either embedded or stand-alone devices. 

[047] The write buffer 482 in each memory interface 470a-d is used to store 

write requests while a read request is being serviced. In such a system, the processor 
104 can issue a write request to a system memory device 440a-d even if the memory 
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device to which the write request is directed is busy servicing a prior write or read 
request. The write buffer 482 preferably accumulates several write requests received 
from the switch 460, which may be interspersed with read requests, and subsequently 
applies them to each of the memory devices 160 in sequence without any intervening 
read requests. By pipelining the write requests in this manner, they can be more 
efficiently processed since delays inherent in read/write turnarounds are avoided. The 
ability to buffer write requests to allow a read request to be serviced can also greatly 
reduce memory read latency since read requests can be given first priority regardless of 
their chronological order. 

[048] The use of the cache memory unit 484 in each memory interface 470a-d 

allows the processor 104 to receive data responsive to a read command directed to a 
respective system memory device 160 without waiting for the memory device 160 to 
provide such data in the event that the data was recently read from or written to that 
memory device 160. The cache memory unit 484 thus reduces the read latency of the 
system memory devices 440a-d to maximize the memory bandwidth of the computer 
system. Similarly, the processor 104 can store write data in the cache memory unit 484 
and then perform other functions while the memory controller 480 in the same memory 
interface 470a-d transfers the write data from the cache memory unit 484 to the memory 
device 160 to which it is coupled. 

[049] Further included in the memory hub local 150 may be a self-test module 

490 coupled to the switch 460 through a test bus 492. The self-test module 490 is 
further coupled to a maintenance bus 496, such as a System Management Bus (SMBus) 
or a maintenance bus according to the Joint Test Action Group (JTAG) and IEEE 
1149.1 standards. Both the SMBus and JTAG standards are well known by those 
ordinarily skilled in the art. Generally, the maintenance bus 496 provides a user access 
to the self-test module 490 in order to set memory testing parameters and receive test 
results. For example, the user can couple a separate PC host via the maintenance bus 
496 to set the relative timing between signals that are applied to the memory devices 
160. Similarly, data indicative of the relative timing between signals that are received 
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from the memory devices 160 can be coupled to the PC host via the maintenance bus 
496. 

[050] Further included in the memory hub 150 may be a DMA engine 486. 

coupled to the switch 460 through a bus 488. The DMA engine 486 enables the 
memory hub 130 to move blocks of data from one location in one of the memory 
devices 160 to another location in the memory device without intervention from the 
processor 104. The bus 488 includes a plurality of conventional bus lines and signal 
lines, such as address, control, data buses, and the like, for handling data transfers in the 
system memory. Conventional DMA operations well known by those ordinarily skilled 
in the art can be implemented by the DMA engine 486. 

[051] From the foregoing it will be appreciated that, although specific 

embodiments of the invention have been described herein for purposes of illustration, 
various modifications may be made without deviating from the spirit and scope of the 
invention. Accordingly, the invention is not limited except as by the appended claims. 
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CLAIMS 

What is claimed is: 

1. A method of operating a memory system having a memory hub 
controller coupled to a plurality of memory hubs each of which includes at least one 
receiver that is initialized before use, the method comprising: 

transmitting a downstream initialization complete signal from the 
memory hub controller to one of the memory hubs when initialization of a receiver in 
the memory hub controller has been completed; 

determining if a respective downstream initialization complete signal is 
being received by each of the memory hubs; 

determining if initialization of at least one receiver in each of the 
memory hubs has been completed; 

if the respective downstream initialization complete signal is being 
received by each of the memory hubs except for a final memory hub furthest 
downstream from the memory hub controller and initialization of at least one receiver in 
the memory hub has been completed, transmitting a downstream initialization complete 
signal from the memory hub; 

if the respective downstream initialization complete signal is being 
received by the final memory hub and initialization of at least one receiver in the final 
memory hub has been completed, transmitting an upstream initialization complete 
signal from the memory hub; 

if the respective upstream initialization complete signal is being received 
by each of the memory hubs determining if an upstream initialization complete signal is 
being received by the memory hub; 

if the respective upstream initialization complete signal is being received 
by each of the memory hubs and initialization of all receivers in the memory hub has 
been completed, transmitting an upstream initialization complete signal from the 
memory hub; and 



; 
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if a respective upstream initialization complete signal is being received 
by the memory hub controller, transitioning the memory system to normal operation. 

2. The method of claim 1, further comprising switching each of the 
memory hubs to a normal operating mode responsive to all of the memory hubs 
transmitting a respective upstream initialization complete signal. 

3. The method of claim 1 wherein each of the initialization 
complete signals comprises the transitioning of a predetermined series of binary values 
to a compliment of the binary values. 

4. The method of claim 3 wherein each of the predetermined series 
of binary values are used by the memory hub controller and the memory hubs initialize 
the receivers in the memory hub controller and the memory hubs. 

5. The method of claim 3 wherein each of the predetermined series 
of binary values and the complement of each of the predetermined series of binary 
values are used by the memory hub controller and the memory hubs initialize the 
receivers in the memory hub controller and the memory hubs. 

6. The method of claim 1 , further comprising: 

transmitting from the memory hub controller and each of the memory 
hubs a respective initialization signal; and 

receiving at the memory hub controller and each of the memory hubs a 
respective one of the initialization signals. 



21 

7. The method of claim 6 wherein each of the initialization 
complete signals comprises a change in the nature of a respective one of the 
initialization signals. 

8. The method of claim 7 wherein each of the initialization signals 
comprises a predetermined series of binary values. 

\ 9. The method of claim 8 wherein the change in the nature of each 
of the initialization signals comprises a transitioning of the predetermined series of 
binary values to a compliment of the predetermined series of binary values. 

10. The method of claim 1 wherein a plurality of the memory hubs 
each comprise an upstream receiver, a downstream transmitter, an upstream receiver 
and an upstream receiver. 

11. In a memory system having at least one memory hub each having 
a downstream receiver, a downstream transmitter, a downstream receiver and an 
upstream receiver, and a memory hub controller having a downstream transmitter and 
an upstream receiver, the memory hub controller having its downstream transmitter 
coupled to the downstream receiver of a first of the memory hubs and its upstream 
receiver coupled to the upstream transmitter of the first of the memory hubs, each of the 
memory hubs hubs having its downstream receiver coupled to the downstream 
transmitter of an upstream one of the memory modules, its upstream transmitter coupled 
to the upstream receiver of the upstream one of the memory modules, each of the 
receivers including an initialization circuit for initializing the respective receiver, a 
method of determining when all of the receivers have been initialized, comprising: 

in the memory hub controller: 
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transmitting a downstream initialization complete signal from the 
downstream transmitter when the initialization of the upstream receiver has been 
completed; 

in each of the memory hubs except for a memory hub furthest 
downstream from the memory hub controller: 

determining if a downstream initialization complete signal is 
being received by the downstream receiver; 

determining if initialization of at least one of the downstream 
receiver and the upstream receiver has been completed; and 

if the downstream initialization complete signal is being received 
by the downstream receiver and initialization of at least one of the downstream 
receiver and the upstream receiver has been completed, transmitting a 
downstream initialization complete signal from the downstream transmitter; 

in the memory hub furthest downstream from the memory hub 

controller: 

determining if a downstream initialization complete signal is 
being received by the downstream receiver; 

determining if initialization of at least the downstream receiver 

has been completed; and 

if the downstream initialization complete signal is being received 
by the downstream receiver and initialization of at least the downstream receiver 
has been completed, transmitting an upstream initialization complete signal from 
the upstream transmitter; 

in each of the memory hubs except for a memory hub furthest 
downstream from the memory hub controller: 

determining if an upstream initialization complete signal is being 
received by the upstream receiver; and 



23 

if the upstream initialization complete signal is being received by 
the upstream receiver and initialization of both the downstream receiver and the 
upstream receiver has been completed, transmitting an upstream initialization 
complete signal from the upstream transmitter; and 
in the memory hub controller: 

determining if an upstream initialization complete signal is being 
received by the upstream receiver; and 

if the upstream initialization complete signal is being received by 
the upstream receiver, determining that all of the receivers in the memory hub 
controllers and all of the memory hubs have been initialized. 

12. The method of claim 11 wherein the act of determining if 
initialization of at least one of the downstream receiver and the upstream receiver has 
been completed comprises determining if initialization of the downstream receiver has 
been completed. 

13. The method of claim 11 wherein the act of determining if 
initialization of at least one of the downstream receiver and the upstream receiver has 
been completed comprises determining if initialization of both the downstream receiver 
and the upstream receiver have been completed. 

14. The method of claim 11, further comprising switching the 
memory hub controller to a normal operating mode responsive to the memory hub 
controller determining that all of the receivers in the memory hub controllers and all of 
the memory hubs have been initialized. 

15. The method of claim 11, further comprising switching each of the 
memory hubs to a normal operating mode responsive to the upstream transmitter in the 
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respective memory hub transmitting a respective upstream initialization complete 
signal 

16. The method of claim 11 wherein each of the initialization 
complete signals comprises the transitioning of a predetermined series of binary values 
to a compliment of the binary values. 

17. The method of claim 16 wherein each of the predetermined series 
of binary values are used by the memory hub controller and the memory hubs initialize 
the receivers in the memory hub controller and the memory hubs. 

1 8. The method of claim 1 6 wherein each of the predetermined series 
of binary values and the complement of each of the predetermined series of binary 
values are used by the memory hub controller and the memory hubs initialize the 
receivers in the memory hub controller and the memory hubs. 

19. The method of claim 1 1, further comprising: 
transmitting from each of the transmitters an initialization signal; and 
receiving at each of the receivers a respective one of the initialization 

signals transmitted by a respective transmitter. 

20. The method of claim 19 wherein each of the initialization 
complete signals comprises a change in the nature of a respective one of the 
initialization signals. 

21 . The method of claim 20 wherein each of the initialization signals 
comprises a predetermined series of binary values. 
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22. The method of claim 21 wherein the change in the nature of each 
of the initialization signals comprises a transitioning of the predetermined series of 
binary values to a compliment of the predetermined series of binary values. 

23. A method of operation a memory hub controller including a 
transmitter and an receiver having an initialization circuit for initializing the receiver, 
the method comprising: 

transmitting an initialization complete signal from the transmitter when 
the initialization of the receiver has been completed; 

determining if the initialization complete signal is being received by the 

receiver; and 

if the initialization complete signal is being received by the receiver, 
transitioning the memory hub controller to a normal operating mode. 

24. The method of claim 23 wherein the received initialization 
complete signal comprises the transitioning of a predetermined series of binary values 
to a compliment of the binary values. 

25. The method of claim 24, further comprising using the 
predetermined series of binary values to initialize the receiver in the memory hub 
controller. 

26. The method of claim 25, further comprising using the 
complement of the predetermined series of binary values to initialize the receiver in the 
memory hub controller. 

27. The method of claim 23, further comprising: 
transmitting from the transmitter a respective initialization signal; 
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receiving at the receiver a respective initialization signal; and 

using the initialization signal received by the receiver to initialize the 

receiver. 

28. The method of claim 27 wherein each of the initialization 
complete signals comprises a change in the nature of a respective one of the 
initialization signals. 

29. The method of claim 28 wherein each of the initialization signals 
comprises a predetermined series of binary values. 

30. The method of claim 29 wherein the change in the nature of each 
of the initialization signals comprises transitioning the respective predetermined series 
of binary values to a compliment of the predetermined series of binary values. 

31. A method of operation a memory hub including an upstream 
transmitter, a downstream transmitter, an upstream receiver, and a downstream receiver, 
the method comprising: 

determining if a downstream initialization complete signal is being 
received by the downstream receiver; 

determining if initialization of at least one of the downstream receiver 
and the upstream receiver has been completed; 

if the downstream initialization complete signal is being received by the 
downstream receiver and initialization of at least one of the downstream receiver and 
the upstream receiver has been completed, transmitting a downstream initialization 
complete signal from the downstream transmitter; 

determining if an upstream initialization complete signal is being 
received by the upstream receiver; and 
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if the upstream initialization complete signal is being received by the 
upstream receiver and initialization of both the downstream receiver and the upstream 
receiver has been completed, transmitting an upstream initialization complete signal 
from the upstream transmitter. 

32. The method of claim 31, further comprising transitioning the 
memory hub to a normal operating mode responsive to the upstream receiver receiving 
the upstream initialization complete signal and initialization completion of both the 
downstream receiver and the upstream receiver. 

33. The method of claim 31 wherein each of the initialization 
complete signals comprises the transitioning of a predetermined series of binary values 
to a compliment of the binary values. 

34. The method of claim 33, further comprising using the 
predetermined series of binary values to initialize the upstream and downstream 
receivers in the memory hub. 

35. The method of claim 34, further comprising using the 
complement of the predetermined series of binary values to initialize the upstream and 
downstream receivers in the memory hub. 

36. The method of claim 3 1 , further comprising: 

transmitting from the upstream transmitters an upstream initialization 

signal; 

transmitting from the downstream transmitter a downstream initialization 

signal; 

receiving at the upstream receiver an upstream initialization signal; 



28 



receiving at the downstream receiver a downstream initialization signal; 
using the upstream initialization signal to initialize the upstream 

receiver; and 

using the downstream initialization signal to initialize the downstream 

receiver. 

37. The method of claim 36 wherein each of the initialization 
complete signals comprises a change in the nature of a respective one of the 
initialization signals. 

38. The method of claim 37 wherein each of the initialization signals 
comprises a predetermined series of binary values. 

39. The method of claim 38 wherein the change in the nature of each 
of the initialization signals comprises transitioning the respective predetermined series 
of binary values to a compliment of the predetermined series of binary values. 

40. A memory system, comprising: 
a memory hub controller, comprising: 

an upstream receiver including an initialization circuit for 
initializing the receiver, the upstream receiver being operable to determine if an 
upstream initialization complete signal is being received by the upstream 
receiver; and 

a downstream transmitter transmitting a downstream initialization 
complete signal when the initialization of the upstream receiver has been 
completed; 

a terminating memory module, comprising: 

a plurality of memory devices; and 
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a memory hub coupled to the plurality of memory devices, the 
memory hub comprising: 

a downstream receiver including an initialization circuit 
for initializing the receiver, the downstream receiver being operable to 
determine if a downstream initialization complete signal is being 
received by the downstream receiver, the downstream receiver further 
being operable to determine if initialization of the downstream receiver 
has been completed; and 

an upstream transmitter transmitting an upstream 
initialization complete signal from the upstream transmitter if the 
downstream initialization complete signal is being received by the 
downstream receiver and initialization of at least the downstream 
receiver has been completed; and 

at least one intermediate memory module, each intermediate memory 
module comprising: 

a plurality of memory devices; and 

a memory hub coupled to the plurality of memory devices, the 
memory hub comprising: 

a downstream receiver coupled to the downstream 
transmitter in either the memory hub controller or in the memory hub of 
another of the intermediate memory modules, the downstream receiver 
including an initialization circuit for initializing the receiver, the 
downstream receiver being operable to determine if a downstream 
initialization complete signal is being received by the downstream 
receiver, the downstream receiver further being operable to determine if 
initialization of at least one of the downstream receiver and the upstream 
receiver has been completed; 
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a downstream transmitter coupled to the downstream 
receiver in the memory hub of either the terminating memory module or 
another of the intermediate memory modules, the downstream 
transmitter being operable to transmit a downstream initialization 
complete signal if the downstream initialization complete signal is being 
received by the downstream receiver and initialization of at least one of 
the downstream receiver and the upstream receiver has been completed; 

an upstream receiver coupled to the upstream transmitter 
in the memory hub of either the terminating memory module or another 
of the intermediate memory modules, the upstream receiver including an 
initialization circuit for initializing the receiver, the upstream receiver 
being operable to determine if an upstream initialization complete signal 
is being received by the upstream receiver; and 

an upstream transmitter coupled to the upstream receiver 
in either the memory hub controller or in the memory hub of another of 
the intermediate memory modules, the upstream transmitter being 
operable to transmit an upstream initialization complete signal if an 
upstream initialization complete signal is being received by the upstream 
receiver and initialization of both the downstream receiver and the 
upstream receiver has been completed. 

41. The memory system of claim 40 wherein the memory devices in 
each of the memory modules comprise dynamic random access memory devices. 

42. The memory system of claim 40 wherein the downstream 
transmitter in the memory hub of each of the intermediate memory modules is operable 
to transmit the downstream initialization complete signal if the downstream 
initialization complete signal is being received by the downstream receiver and 
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initialization of the downstream receiver but not necessarily the upstream receiver has 
been completed. 

43. The memory system of claim 40 wherein the downstream 
transmitter in the memory hub of each of the intermediate memory modules is operable 
to transmit the downstream initialization complete signal if the downstream 
initialization complete signal is being received by the downstream receiver and 
initialization of both the downstream receiver and the upstream receiver has been 
completed. 

44. The memory system of claim 40 wherein the memory hub 
controller is further operable to switch the memory hub controller to a normal operating 
mode responsive to the receiver in the memory hub controller receiving the upstream 
initialization complete signal. 

45. The memory system of claim 40 wherein each of the initialization 
complete signals comprises the transitioning of a predetermined series of binary values 
to a compliment of the binary values. 

46. The memory system of claim 45 wherein the receivers in the 
memory hub controller and in each of the memory hubs are operable to use a respective 
one of the predetermined series of binary values to initialize the receivers. 

47. The memory system of claim 45 wherein the receivers in the 
memory hub controller and in each of the memory hubs are operable to use a respective 
one of the predetermined series of binary values and the complement of the respective 
one of the predetermined series of binary values to initialize the receivers. 
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48. The memory system of claim 40 wherein each of the transmitters 
is further operable to transmit a respective initialization signal, and each of the receivers 
is operable to receive a respective one of the initialization signals transmitted by a 
respective transmitter. 

49. The memory system of claim 48 wherein each of the initialization 
complete signals comprises a change in the nature of a respective one of the 
initialization signals. 

50. The memory system of claim 49 wherein each of the initialization 
signals comprises a predetermined series of binary values. 

5 1 . The memory system of claim 50 wherein the change in the nature 
of each of the initialization signals comprises a transitioning of the predetermined series 
of binary values to a compliment of the predetermined series of binary values. 

52. A memory module, comprising: 
a plurality of memory devices; and 

a memory hub coupled to the plurality of memory devices, the memory 
hub comprising: 

a downstream receiver including an initialization circuit for 
initializing the receiver, the downstream receiver being operable to determine if 
a downstream initialization complete signal is being received by the downstream 
receiver, the downstream receiver further being operable to determine if 
initialization of the downstream receiver has been completed; 

a downstream transmitter coupled to the downstream receiver in 
the memory hub, the downstream transmitter being operable to transmit a 
downstream initialization complete signal if the downstream initialization 
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complete signal is being received by the downstream receiver and initialization 
of the downstream receiver has been completed; 

an upstream receiver including an initialization circuit for 
initializing the receiver, the upstream receiver being operable to determine if an 
upstream initialization complete signal is being received by the upstream 
receiver; and 

an upstream transmitter coupled to the upstream receiver, the 
upstream transmitter being operable to transmit an upstream initialization 
complete signal if an upstream initialization complete signal is being received 
by the upstream receiver and initialization of both the downstream receiver and 
the upstream receiver has been completed. 

53. The memory module of claim 52 wherein the memory devices 
comprise dynamic random access memory devices. 

54. The memory module of claim 52 wherein the downstream 
transmitter is operable to transmit the downstream initialization complete signal if the 
downstream initialization complete signal is being received by the downstream receiver 
and initialization of both the downstream receiver and the upstream receiver has been 
completed. 

55. The memory module of claim 52 wherein each of the 
initialization complete signals comprises the transitioning of a predetermined series of 
binary values to a compliment of the binary values. 

56. The memory module of claim 55 wherein the receivers are 
operable to use a respective one of the predetermined series of binary values to initialize 
the receivers. 
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57. The memory module of claim 55 wherein the receivers are 
operable to use a respective one of the predetermined series of binary values and the 
complement of the respective one of the predetermined series of binary values to 
initialize the receivers. 

58. The memory module of claim 52 wherein each of the transmitters 
is further operable to transmit a respective initialization signal, and each of the receivers 
is operable to receive a respective one of the initialization signals. 

59. The memory module of claim 58 wherein each of the 
initialization complete signals comprises a change in the nature of a respective one of 
the initialization signals. 

60. The memory module of claim 59 wherein each of the 
initialization signals comprises a predetermined series of binary values. 

61. The memory module of claim 60 wherein the change in the 
nature of each of the initialization signals comprises a transitioning of the 
predetermined series of binary values to a compliment of the predetermined series of 
binary values. 

62 . A processor-based system, comprising: 
a processor having a processor bus; 

a system controller coupled to the processor bus, the system controller 
having a peripheral device port; 

at least one input device coupled to the peripheral device port of the 
system controller; 
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at least one output device coupled to the peripheral device port of the 
system controller; 

at least one data storage device coupled to the peripheral device port of 
the system controller; and 

a first upstream data bus; 
a first downstream data bus; 

a memory hub controller coupled to the processor bus, the memory hub 
controller comprising: 

an upstream receiver including an initialization circuit for 
initializing the receiver, the upstream receiver being operable to determine if an 
upstream initialization complete signal is being received by the upstream 
receiver; and 

a downstream transmitter transmitting a downstream initialization 
complete signal when the initialization of the upstream receiver has been 
completed; 

a terminating memory module, comprising: 
a plurality of memory devices; and 

a memory hub coupled to the plurality of memory devices, the 
memory hub comprising: 

a downstream receiver including an initialization circuit 
for initializing the receiver, the downstream receiver being operable to 
determine if a downstream initialization complete signal is being 
received by the downstream receiver, the downstream receiver further 
being operable to determine if initialization of the downstream receiver 
has been completed; and 

an upstream transmitter transmitting an upstream 
initialization complete signal from the upstream transmitter if the 
downstream initialization complete signal is being received by the 
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downstream receiver and initialization of at least the downstream 
receiver has been completed; and 

at least one intermediate memory module, each intermediate memory 
module comprising: 

a plurality of memory devices; and 

a memory hub coupled to the plurality of memory devices, the 

memory hub comprising: 

a downstream receiver coupled to the downstream 
transmitter in either the memory hub controller or in the memory hub of 
another of the intermediate memory modules, the downstream receiver 
including an initialization circuit for initializing the receiver, the 
downstream receiver being operable to determine if a downstream 
initialization complete signal is being received by the downstream 
receiver, the downstream receiver further being operable to determine if 
initialization of at least one of the downstream receiver and the upstream 
receiver has been completed; 

a downstream transmitter coupled to the downstream 
receiver in the memory hub of either the terminating memory module or 
another of the intermediate memory modules, the downstream 
transmitter being operable to transmit a downstream initialization 
complete signal if the downstream initialization complete signal is being 
received by the downstream receiver and initialization of at least one of 
the downstream receiver and the upstream receiver has been completed; 

an upstream receiver coupled to the upstream transmitter 
in the memory hub of either the terminating memory module or another 
of the intermediate memory modules, the upstream receiver including an 
initialization circuit for initializing the receiver, the upstream receiver 
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being operable to determine if an upstream initialization complete signal 
is being received by the upstream receiver; and 

an upstream transmitter coupled to the upstream receiver 
in either the memory hub controller or in the memory hub of another of 
the intermediate memory modules, the upstream transmitter being 
operable to transmit an upstream initialization complete signal if an 
upstream initialization complete signal is being received by the upstream 
receiver and initialization of both the downstream receiver and the 
upstream receiver has been completed. 

63. The processor-based system of claim 62 wherein the memory 
devices in each of the memory modules comprise dynamic random access memory 
devices. 

64. The memory system of claim 62 wherein the downstream 
transmitter in the memory hub of each of the intermediate memory modules is operable 
to transmit the downstream initialization complete signal if the downstream 
initialization complete signal is being received by the downstream receiver and 
initialization ofVthe downstream receiver but not necessarily the upstream receiver has 
been completed.; 

65. The memory system of claim 62 wherein the downstream 
transmitter in the memory hub of each of the intermediate memory modules is operable 
to transmit the downstream initialization complete signal if the downstream 
initialization complete signal is being received by the downstream receiver and 
initialization of both the downstream receiver and the upstream receiver has been 
completed. 
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66. The memory system of claim 62 wherein the memory hub 
controller is further operable to switch the memory hub controller to a normal operating 
mode responsive to the receiver in the memory hub controller receiving the upstream 
initialization complete signal. 

67. The memory system of claim 62 wherein each of the initialization 
complete signals comprises the transitioning of a predetermined series of binary values 
to a compliment of the binary values. 

68. The memory system of claim 67 wherein the receivers in the 
memory hub controller and in each of the memory hubs are operable to use a respective 
one of the predetermined series of binary values to initialize the receivers. 

69. The memory system of claim 67 wherein the receivers in the 
memory hub controller and in each of the memory hubs are operable to use a respective 
one of the predetermined series of binary values and the complement of the respective 
one of the predetermined series of binary values to initialize the receivers. 

70. The memory system of claim 62 wherein each of the transmitters 
is further operable to transmit a respective initialization signal, and each of the receivers 
is operable to receive a respective one of the initialization signals transmitted by a 
respective transmitter. 

7 1 . The memory system of claim 70 wherein each of the initialization 
complete signals comprises a change in the nature of a respective one of the 
initialization signals. 
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72. The memory system of claim 7 1 wherein each of the initialization 
signals comprises a predetermined series of binary values. 

73. The memory system of claim 72 wherein the change in the nature 
of each of the initialization signals comprises a transitioning of the predetermined series 
of binary values to a compliment of the predetermined series of binary values. 



SYSTEM AND METHOD FOR COMMUNICATING THE 
SYNCHRONIZATION STATUS OF MEMORY MODULES DURING 
INITIALIZATION OF THE MEMORY MODULES 

ABSTRACT OF THE DISCLOSURE 

A memory system includes a memory hub controller coupled to a 
plurality of memory modules each of which includes a memory hub. The memory hub 
controller and the memory hubs each include at least one receiver that is synchronized 
to an internal clock signal during initialization. The memory hub controller and the 
memory hubs each transmit an initialization complete signal downstream when at least 
one receiver in the controller or hub is initialized and, in the case of the memory hubs, 
when a downstream initialization signal has also been received. Similarly, the memory 
hubs transmit an initialization signal upstream to another memory hub or the controller 
when both of its receivers are initialized and an upstream initialization signal has also 
been received. Receipt of an upstream initialization signal by the memory hub 
controller signifies that all of the receivers have been initialized. 
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